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Abstract 

Command  World  is  an  unclassified  simulation  used  to  create  data  for  DARPA’s 
Personalized  Assistant  that  Learns  (PAL)  program.  The  Command  World  scenario  was 
expressly  designed  as  a  crisis  action  planning  exercise  in  order  to  replicate  the 
communications,  collaboration,  and  information  requirements  inherent  in  a  military 
domain  and  to  facilitate  realistic  and  relevant  communications  and  collaboration 
exchanges  across  an  information  system  network.  The  development  of  a  robust  and 
unclassified  scenario  and  all  of  the  supporting  training  artifacts  was  integral  to  the 
program  goals  of  process  execution,  transfer  learning,  and  the  eventual  development  of  a 
bootstrap  ontology  for  PAL  use  in  a  military  environment.  The  PAL  program  is  designed 
to  explore  the  possibility  of  creating  tools  that  will  assist  military  planners  in  their  tasks 
through  the  use  of  digital  “assistants.”  By  instrumenting  the  environment  over  5 
experiments,  over  200,000  events  were  captured.  This  paper  will  describe  the 
environment,  the  data  collected,  and  how  the  data  is  being  used. 
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Introduction 


Defense  Advanced  Research  Projects  Agency  (DARPA)  has  initiated  the  Personalized 
Assistant  that  Learns  (PAL)  program  -  a  cognitive  technology  program  that  offers 
significant  value  not  only  to  the  military  user  community,  but  also  to  business  and 
academic  sectors.  The  project  aim  is  to  spur  pioneering  research  in  cognitive  information 
processing  -  including  areas  of  artificial  intelligence  (AI),  machine  learning,  knowledge 
representation  and  reasoning,  machine  perception,  natural  language  processing,  and 
behavioral  studies.  DARPA  envisions  PAL  as  an  “enduring  personalized  cognitive 
assistant”  to  help  decision-makers  manage  their  complex  worlds  of  multiple  simultaneous 
tasks  and  unexpected  situations.  [Gunning  2004] 

PAL’s  original  focus  was  as  an  office  assistant  tasked  with  scheduling  meetings,  handling 
correspondence  and  assisting  with  quarterly  reports.  The  concept  behind  PAL  technology 
was  to  lend  assistance  by  observing  and  interacting  with  office  workers.  Commercial 
software,  e-mail  and  scheduling  programs  will  be  adapted  for  PAL  purposes.  PAL  will 
augment  these  functions  by  adding  modules  that  will  train  the  software  to  recognize 
users’  preferences,  and  by  adding  components  that  will  decide  when  to  interrupt  the  user 
with  questions.  DARPA  hopes  the  research  will  lead  to  systems  that  can  reason,  learn 
from  experience,  take  advice  and  respond  intelligently  to  new  situations. 

Command  World  Simulation  Exercise 

In  conjunction  with  the  Naval  Postgraduate  School  (NPS),  Space  and  Naval  Warfare 
Systems  Center  San  Diego  (SSC  San  Diego)  was  tasked  to  develop  the  Command  World 
for  performing  evaluations  of  PAL  technology  in  support  of  transition  efforts.  One  of  the 
Command  World  objectives  is  to  develop  military  simulation  exercises  to  support  data 
collection,  initially  without  PAL  software,  and  then  with  increasingly  functional,  iterative 
versions  of  PAL  software,  in  order  to  support  transition  of  PAL  technologies  to  a  military 
environment.  SSC  San  Diego  with  NPS  conducted  a  series  of  Command  World 
exercises  -  with  each  exercise  testing  and  evaluating  improved  PAL  functionality  since 
September  2004. 

“Operation  CROWN  JEWEL”  Scenario 

Operation  CROWN  JEWEL  is  a  simulation  for  producing  data  for  the  Command  World 
project.  This  data  is  a  critical  component  of  assessing  the  effectiveness  of  software 
assistants  to  support  planning  and  coordination  activities.  The  intent  of  Operation 
CROWN  JEWEL  is  to  develop  baseline  data  relative  to  the  IM  tools  used  by  joint 
planners  in  fulfilling  their  planning  responsibilities.  DARPA  will  use  this  data  to  support 
the  CALO  project.  CALO  will  explore  the  possibility  of  creating  tools  that  will  assist 
military  planners  in  their  tasks  through  the  use  of  digital  “assistants.”  Since  these 
assistants  must  “understand”  the  planner’s  tasks  and  how  the  planner  accomplishes  them, 
Command  World  contractors  conduct  a  series  of  simulations  where  these  tasks  are 
studied  and  parameterized.  Accordingly,  Operation  CROWN  JEWEL  is  a  laboratory  for 
this  effort. 

The  simulation  has  a  set  of  participants,  each  with  organizational  roles  and 
responsibilities  related  to  the  planning  and  preparatory  work  supporting  a  hypothetical 
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military  operation.  The  scenario  and  all  related  data  are  unclassified.  The  scenario 
assumes  that  participants  are  members  of  a  Combatant  Commander  (COCOM)  staff 
assigned  planning  responsibilities  for  an  anticipated  peacekeeping  operation  in  a  fictitious 
world  region.  Participants  use  standard  Crisis  Action  Planning  (CAP)  procedures  in 
accomplishing  the  tasks  assigned  during  the  scenario. 

Participants  form  a  joint  planning  cell  at  the  (COCOM)  headquarters.  The  planning 
team’s  objective  is  to  develop  the  Commander’s  Assessment  and,  based  on  the  scenario 
“trigger  events,”  to  conduct  a  CAP  approach  and  develop  a  set  of  Course  of  Action 
(CO A)  alternatives  for  executing  the  CROWN  JEWEL  mission.  The  exercise  scenario 
includes  a  full  range  of  peacekeeping  operations  (PKO),  including  monitoring  treaty 
agreements,  liaison  with  the  respective  belligerent  nation  leaders  (i.e.,  TOPAZ,  CORAL), 
and  conducting  humanitarian  relief  operations.  This  final  task  could  become  extensive  in 
view  of  a  resultant  refugee  problem  and  will  require  support  of  various  International 
Organizations  (10)  and  Non-Govemmental  Organizations  (NGO),  an  extensive  mine 
clearance  and  eradication  operation,  food  and  medical  assistance,  and  security  operations. 

The  Exercise  Test  Director  manages  Operation  CROWN  JEWEL,  and  introduces  events 
at  various  times  thus  affecting  one  or  more  actions.  The  participants  did  not  know  the 
nature  and  timing  of  these  events  beforehand.  The  Exercise  Test  Director  is  also 
responsible  for  sending  email  from  (fictitious)  non-participants.  In  accordance  with 
standard  planning  procedures  at  most  joint  commands,  the  Test  Director  provides  an 
exercise  Battle  Rhythm  to  guide  participants.  Actions  taken  during  the  simulation  must 
be  within  the  confines  of  the  exercise  Battle  Rhythm  and  under  the  direction  of  the 
Exercise  Test  Director  who  functions  as  the  supervisor  of  the  planning  and  event 
execution  effort. 

Data  results  from  conducting  simulation  sessions  where  military  officers  play  participant 
roles  using  instrumented  email,  an  online  calendar,  text  editor,  web  browsers  and  other 
software  to  communicate  with  each  other,  to  collect  information  needed  for  specific 
tasks,  and  to  produce  documents.  Each  participant  has  a  workstation  dedicated  to  the 
simulation,  but  with  no  connection  to  the  participant’s  real  identity  or  any  other  private 
data.  In  this  fashion,  a  record  of  all  simulation  activity  can  be  captured  and  distributed 
without  privacy  concerns.  This  captured  data  includes  email  activity,  calendar  edits,  files 
created  and  edited  and  directory  and  subdirectory  structures  created,  web  browsing 
activity,  etc. 

Participants  also  engage  in  work  activities  related  to  their  normal  billet  assignments  (e.g. 
admin  routines  such  as  travel,  etc.,  web  search  on  status  of  cities,  etc.)  as  well  as  duties  as 
part  of  the  CAP  planning  staff  This  requires  participants  to  produce  some  routine  reports 
at  specified  times,  and  to  complete  other  tasks  related  to  their  role-play  assignments.  In 
some  cases,  participants  collect  information  and  solve  problems,  thereby  requiring  them 
to  exhibit  realistic  web,  calendar,  and  file  searching  actions  as  part  of  the  data  to  be 
collected.  Participants  create  file  and  directory  structures  as  appropriate  for  organizing 
their  tasks.  To  minimize  session  “real  time,”  participants  have  basic  data  (documents, 
spreadsheets,  etc.)  to  use  in  the  performance  of  work  activities. 

Simulation  Domain 
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The  scenario  utilized  in  this  exercise  is  from  an  actual  USJFCOM  J7  unclassified 
exercise,  modified  to  meet  the  data  collection  requirements  best  suited  to  test  the 
capabilities  of  CALO  during  an  operational  planning  event. 

The  knowledge  domain  for  Operation  CROWN  JEWEL  in  the  scenario  simulation  is  the 
CAP  process  at  a  COCOM  headquarters.  Additionally,  the  knowledge  domain  will 
encompass  a  small  set  of  routine  daily  tasks  that  must  be  balanced  with  CAP  duties  by 
the  participants.  The  scenario  and  sequence  of  events  are  intended  as  generic  in  nature 
and  are  not  intended  to  be  specific  to  any  particular  unified  command  or  unified 
command  geographic  region.  Moreover,  the  CROWN  JEWEL  scenario  is  unclassified 
and  is  not  intended  to  represent  any  particular  LF.S.  operational  military  plan  or  potential 
real  world  crisis.  Rather,  the  scenario  is  intended  to  reflect  the  operational  issues  that 
would  confront  a  COCOM  planning  staff  tasked  with  preparing  for  joint  or  multi-national 
force  operations  in  its  assigned  area  of  responsibility  (AOR). 


Figure  1:  Map  of  the  Island  of  Island  nations  of  GEM 


JOINT  PLANNING  TEAM  STRUCTURE 

The  planning  team  is  composed  of  six  military  officers  with  experience  in  joint  planning. 
This  will  include  one  intelligence  planner,  one  logistics  planner,  and  three  joint  force 
planners  (i.e..  Land  Component  Planner,  Maritime  Component  Planner,  Air  Component 
Planner),  preferably  one  from  each  of  the  major  services,  U.S.  Army,  U.S.  Navy,  U.S.  Air 
Force.  The  senior  member  of  the  team  will  be  the  Joint  Planning  Group  Director  who 
will  act  as  both  the  senior  member  of  the  team  and  exercise  mediator  among  the  non¬ 
participant  roles  and  corresponding  game  cues. 
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Operation  CROWN  JEWEL 
Joint  Planning  Group  Structure 


MAJOR  TASKS 

The  planning  team  will  produce  the  information  products  generally  required  at  a 
COCOM  headquarters  during  routine  operations  as  a  crisis  develops.  During  Operation 
CROWN  JEWEL,  the  participants  will  accomplish  the  essential  step  in  the  first  three 
phase  of  CAP  -  Situation  Development,  Crisis  Assessment,  and  COA  Development  by 
accomplishing  the  following: 

•  Review  of  incoming  intelligence  and  other  source  data 

•  Commence  CAP  process  for  a  PKO  under  COCOM  direction 

•  Prepare  and  submit  a  daily  Commander’s  briefing  (using  slideware)  to  COCOM; 
prepare  a  daily  operations  summary  message  (OPSUM) 

•  Respond  to  increasing  exercise  tempo  as  Operation  CROWN  JEWEL  begins 


During  Operation  CROWN  JEWEL,  the  participants  will  produce  the  following  products: 
•  Daily  Commander’s  briefing,  OPSUM 
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•  Commander’s  Assessment  message 

•  A  mission  analysis  based  on  Chairman,  Joint  Chiefs  of  Staff  (C/JCS)  Warning  Order 
(WARNORD)  COA’s  (e.g.,  three  alternative  plans)  for  executing  the  mission 
assigned 

•  Briefing  on  recommended  COA’s 

•  Major  triggering  events 

•  General  situational  data  on  scenario 

•  “Spot”  intelligence  reports  as  scenario  develops 

•  Own  force  data  -  disposition,  location,  readiness 

•  Updated  information  from  open,  non-military  sources  -  lO’s,  NGOs,  Media  (e.g., 
CNN),  etc. 

•  Direction  from  higher  authority  -  intentions,  deadlines,  requests  for  intelligence  / 
information  (RFI’s) 

•  Planned  duration 

This  simulation  spreads  four  weeks  of  simulated  time  over  a  predetermined  number  of 
days  of  actual  data  collection.  Depending  upon  the  desired  Battle  Rhythm,  the  Exercise 
Test  Director  may  choose  to  conduct  the  simulation  over  a  period  anywhere  from  2.5 
contiguous  days,  to  8  days  of  intermittent  exercise  play.  The  Exercise  Test  Director  will 
control  the  time  allotted  to  specific  tasks  in  the  CAP  process  with  a  simulated  Battle 
Rhythm.  This  causes  the  participants  to  accomplish  specific  steps  in  the  CAP  process 
while  under  the  simulated  conditions  of  routine  AOR  operations  (exercise  scenario  start) 
building  to  a  large-scale  multi-national  force  (MNF)  operation.  As  the  scenario 
simulation  progresses  it  is  expected  that  participants  will  gradually  shift  individual 
orientation  and  task  accomplishment  to  group  collaboration  and  integrated  task 
accomplishment  required  by  the  exercise  rigor  imposed  by  the  nature  of  the  scenario 
operation. 

SCENARIO  PRODUCTS 

Products  resulting  from  this  scenario  consisted  mainly  of  emails,  MS  Word  Documents, 
MS  Excel  Spreadsheets  and  MS  PowerPoint  presentations.  These  products  resulted  from 
the  script  stimuli  and  the  form  and  format  of  the  response  data  was  completely  at  the 
discretion  of  the  participant.  The  form  and  format  of  the  products  resulted  from  the 
participants’  comfort  levels  with  the  available  applications  and  the  time  allotted  to  the 
task.  Specific  products  produced  from  this  joint  operation  planning  exercise  include: 

•  Operations  Brief  (daily):  Status  update  on  force  disposition  and  availability 

•  Operation  Summary  (daily):  Status  update  on  force  readiness 

•  Commander’s  Assessment  (Phase  1):  Situational  Development  based  upon  nature  of 
crisis,  intelligence  reports,  force  readiness,  refining  mission  concept  (e.g.  factors  of 
space,  time,  and  resources),  and  preliminary  COA 

•  Commander’s  Estimate  (Phase  III):  Mission  statement.  Situation  Update  COA 
(three),  COA  constraints  and  additional  planning  concerns 
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•  “Free  play”  products:  Administrative  requests,  requests  for  information  (RFI) 
germane  to  gameplay 

Experimentation 

The  Command  World  Exercises  were  conducted  in  the  Command  World  lab  at  NFS. 

Each  participant  was  provided  with  a  notebook  computer  networked  to  a  central  server 
where  events  logged  into  a  database  in  the  Timeline  server.  The  server  also  provided  the 
participants  with  Internet  access,  Microsoft  Exchange,  and  other  web  services. 

All  the  laptops,  servers  and  users,  were  connected  via  Ethernet  behind  (on  the  LAN  side 
of)  the  NetGear  router/switch.  Firewall  protection  was  provided  on  the  Internet  side  of 
the  NetGear  by  the  NFS  network.  Two  of  the  user  laptops  and  the  server  were  connected 
directly  to  the  local  LAN  ports  on  the  back  of  it  using  network  cables.  Another  network 
cable  was  connected  from  the  NetGear  to  a  hub.  The  remaining  user  laptops  were 
connected  to  the  hub.  The  laptops  could  browse  the  WWW  subject  to  any  restrictions 
imposed  by  the  NFS  network  that  we  used  for  connectivity  to  the  Internet. 

After  the  initial  training  briefings  in  the  morning  session  of  the  first  day,  the  Exercise 
Test  Director  provided  the  participants  with  simulation  stimuli  to  generate  responses  that 
required  the  participants  to  use  the  hardware  and  software  tools  provided  by  the  exercise 
principals.  This  offered  an  opportunity  to  familiarize  with  some  of  the  non-standard 
keystrokes  required  to  log  some  events,  such  as  saving  email  attachments  to  the 
participants’  hard  drives.  Farticipants  had  the  opportunity  to  ask  questions  and  to  clarify 
issues  that  would  be  important  throughout  the  conduct  of  the  exercise. 

SRI  personnel  provided  NFS  personnel  and  the  participants  with  training  pertaining  to  the 
CALO  instrumentation.  Dedicated  NFS  personnel  built  and  tested  the  experiment 
laboratory  at  the  NFS  and  assisted  SRI  and  the  participants  throughout  the  exercise  with 
network  and  system  administration  support.  NFS  and  SRI  personnel  also  monitored  and 
archived  all  instrumented  data  upon  completion  of  the  exercise. 

Data  Collection 

Each  participant  notebook  computer  hosted  a  client  that  logged  all  MS  Office  activities  to 
a  central  database  on  the  networked  CALO  server  via  the  following  software  modules. 

FileSystemWatcher  (FSW):  Monitors  the  file  system  and  log  events  when  file  system 
changes.  The  events  are  logged  to  a  file  in  the  CALO  Timeline  Server  that  records  all 
activity  in  My  Documents  and  the  subfolders. 

Internet  Explorer  (IE)  Flug-in:  Captures  user  actions  in  IE  and  Microsoft  file  Explorer.  It 
logs  navigation  and  events  to  the  CALO  Timeline  Server  for  indexed  archival. 

MS  Outlook:  Captures  relatively  "fine-grained"  instrumentation  of  email,  calendar, 
appointments,  and  contact  transactions  by  users  of  MS  Outlook.  It  logs  these  transactions 
as  CALO  events  and  writes  these  events  to  a  local  file  and/or  sends  them  to  the  CALO 
timeline  server  for  indexed  archival.  In  this  release  of  CALO  software,  file  attachments 
must  be  saved  using  the  CALO  drop-down  menu  from  the  main  page. 

Outlook  Flug-In:  Captures  email,  contact  and  appointment  events  in  Outlook  and  logs  the 
event  data  to  the  local  file  system  and  to  the  TimeLine  Server  via  OAA  2.3.0. 
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Timeline  Server:  Archives  and  indexes  CALO  events  sent  by  FileSystemWatcher  and 
MS  Outlook  Mail  via  OAA  2.3.0.  and  provides  search  and  subscription  capabilities 
across  this  data. 

Timeline  Viewer:  A  simple  GUI  for  searching  events  stored  in  the  Timeline  Server. 

Each  participant  performed  tasks  associated  with  the  real-world  staff  position  they 
represented.  All  participant  tasks  supported  the  Joint  Crisis  Action  Planning  Model  used 
by  US  Military  personnel  assigned  to  Joint  Planning  staffs. 

It  was  critical  to  exercise  realism  that  the  participants  provided  a  number  of  ad-lib  events, 
all  unknown  to  the  other  participants  prior  to  their  injections  into  the  scenario.  For 
example,  J2  injected  an  ad-lib  event  by  reporting  a  captured  TOPAZ  soldier  carried  a 
chemical  protective  mask  and  nerve  gas  antidote.  This  generated  the  appropriate  message 
traffic  that  would  warn  peacekeepers  of  a  potential  chemical  agent  or  biological  threat. 
Later,  J2  again  injected  an  ad-lib  event  indicating  the  simultaneous  detonation  of  car 
bombs  near  peacekeeping  bases. 

Each  participant  injected  events  pertaining  to  their  area  of  expertise.  J4  reported  the  crash 
of  an  aircraft  carrying  medical  supplies,  while  N3  reported  treacherous  sea  states  caused 
by  the  proximity  of  a  hurricane.  These  and  other  events  represent  the  type  of  unplanned 
stimuli  that  would  generate  a  flurry  of  activity  throughout  the  staff. 

Experimentation  Results 

The  exercise  provided  enhancement  to  coherent  and  logical  information  transmission 
over  information  system  network  along  with  enhancement  to  processing  and  storage  of 
information  over  information  system. 

Outlook  was  the  primary  software  tool  used  by  the  participants  during  the  exercise. 
According  to  the  participants,  their  use  of  the  available  software  tools  broke  out  as 
follows: 

The  collected  data  from  each  exercise  is  summarized  in  the  following  tables. 


Command  World  Simulation  Exercise  #1  Event  Data  Collection 


Event  Type 

Thru 

7/26 

7/27 

7/28 

Totals 

OutlooklA  TimeLineSearch 

171 

727 

664 

1391 

OutlooklA 

3971 

16288 

15116 

31404 

lEPLUGIN 

1171 

847 

1280 

2127 

File  Events 

1156 

3609 

3080 

6689 

Totals: 

6771 

21471 

20140 

41611 
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Command  World  Simulation  Exercise  #2  Event  Data  Collection 


Event  Type 

9/14 

9/15 

9/16 

Totals 

OutlooklA  TimeLineSearch 

98 

360 

772 

1230 

OutlooklA 

3121 

13888 

28664 

45673 

lEPLUGIN 

469 

552 

2185 

3206 

File  Events 

318 

540 

1282 

2140 

Totals: 

4006 

15340 

32903 

52249 

Command  World  Simulation  Exercise  #3  Event  Data  Collection 


Event  Type 

1/26 

1/28 

1/31 

2/2 

Totals 

Calendar: 

74 

48 

100 

88 

310 

Web  Browser: 

1803 

1786 

1291 

1001 

5881 

File  Browser: 

36 

137 

27 

3300 

3500 

E-mail: 

3904 

2093 

3162 

2438 

11597 

Chat: 

610 

757 

623 

534 

2524 

Track: 

41 

7 

0 

1736 

1784 

Task  Start: 

252 

154 

237 

241 

884 

Other: 

1035 

877 

788 

858 

3558 

Totals: 

7755 

5859 

6228 

10196 

30038 

Command  World  Simulation  Exercise  #4  Event  Data  Collection 


Event  Type 

3/9 

3/11 

3/14 

3/16 

Totals 

Calendar: 

57 

60 

106 

135 

358 

Web  Browser: 

1490 

4940 

1817 

1558 

9805 

File  Browser: 

223 

0 

478 

262 

963 

E-mail: 

8968 

6420 

3802 

5547 

24737 

Chat: 

619 

641 

562 

580 

2402 

Track: 

163 

757 

295 

833 

2048 

Task  Start: 

110 

88 

180 

246 

624 

Data  Applications: 

48 

48 

72 

99 

267 

CALO  Query: 

0 

0 

4 

27 

31 

Change  Applications: 

756 

348 

666 

892 

2662 

Totals: 

12434 

13302 

7982 

10179 

43897 

Command  World  Simulation  Exercise  #5  Event  Data  Collection 
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Event  Type 

4/12 

4/13 

4/14 

Totals 

Calendar: 

204 

397 

143 

744 

Web  Browser: 

3103 

4407 

1677 

9187 

File  Browser: 

807 

1401 

96 

4104 

E-mail: 

10336 

21065 

9216 

38817 

Chat: 

1328 

3480 

2479 

7287 

Track: 

928 

3038 

747 

4713 

Task  Start: 

96 

202 

62 

360 

Data  Applications: 

74 

159 

71 

304 

CALO  Query: 

0 

7 

0 

7 

Change  Applications: 

885 

1338 

632 

2855 

Totals: 

17761 

35494 

15123 

68378 

Use  of  Data  in  Learning  Capability  Development 

One  of  the  capabilities  being  developed  in  the  CALO  program  is  process  execution 
assistance.  CALO  helps  the  user  to  perform  tasks.  In  order  for  CALO  to  do  this, 
observation  of  the  user’s  activities  is  necessary.  To  test  CALO  learning  algorithms, 
developers  will  be  able  to  use  Command  World  data  just  as  if  their  modules  were 
monitoring  the  progress  of  the  exercise  in  real  time. 

Many  of  the  capabilities  for  CALO  are  designed  for  learning.  Most  of  the  testing  is  being 
done  in  an  office  automation  environment  where  events  are  plentiful  in  a  real  world 
environment  without  adding  significant  risk.  However,  this  means  that  CALO  is  learning 
about  a  different  environment  than  we  are  targeting.  Another  use  of  the  Command  World 
data  is  to  compare  the  behavior  of  CALO  when  provided  events  from  the  military 
environment  and  indeed  determine  if  the  learning  being  done  in  the  office  environment  is 
benefiting  use  later  in  the  military  environment.  These  tests  of  transfer  learning  were 
performed  in  the  last  iteration  of  Command  World  by  researchers  from  MIT  and  Oregon 
State  University  with  the  assistance  of  NFS  and  SSC  SD  personnel.  Some  of  the  results 
are  published  in  [Marx+  2005]. 

The  final  use  of  the  data  being  collected  is  the  eventual  development  of  a  bootstrap 
ontology  for  PAL  use  in  a  military  environment.  By  collecting  events  and  the  associated 
data,  we  have  data  that  will  support  knowledge  of  a  crisis  action  planning  situation  in  a 
command  staff  environment. 

Summary 

In  support  of  data  collection  for  the  CALO  leaning  algorithm  development,  the 
Command  World  exercises,  were  conducted  over  eleven  days,  generated  236,173  logged 
events.  The  exercise  was  a  success  in  that  it  demonstrated  that  the  scenario  scripts  were 
sufficient  for  the  participants  to  perform  the  essential  CAP  tasks,  and  collected  data  that 
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will  be  useful  for  further  study.  It  also  provided  realistic  baseline  data  from  a  military 
domain  for  the  test  and  evaluation  of  CALO  software,  once  learning  capabilities  are 
sufficiently  mature  to  run  further  tests.  The  baseline  data  will  be  useful  in  comparing 
those  later  results  in  determining  whether  efficiency  and  effectiveness  were  improved 
through  learning.  We  had  the  benefits  of  having  reservists  as  exercise  participants  who 
brought  in  their  experience  from  both  the  military  and  technical  environments  as  they 
work  in  technical  positions  in  their  civilian  careers.  Their  professionalism,  which  was 
demonstrated  by  freely  played  their  roles  in  the  simulation,  contributed  to  the  success  of 
generating  large  quantity  of  high  quality  data. 

The  collected  data  is  releasable  to  researchers  within  the  government  and  those  under 
contract  with  the  government  for  relevant  research. 

References 
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Introduction 
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DARPA  PAL  Program 
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Personalized  Assistant  that  Learns  (PAL)  Program 

-  PAL  program  is  designed  to  explore  the  possibility  of  creating 
tools  that  will  assist  military  planners  in  their  tasks  through  the 
use  of  digital  “assistants” 

-  Spur  pioneering  research  in  cognitive  information  processing  - 
including  areas  of  artificial  intelligence  (Al),  machine  learning, 
knowledge  representation  and  reasoning,  machine 
perception,  natural  language  processing,  and  behavioral 
studies 

-  Administered  by  numerous  educational  institutions,  evaluates 
two  domains  (civilian  and  military)  and  gathers  usable  data 
from  each  to  achieve  the  project  goals;  COMMAND  WORLD 
focus  on  the  military  domain 


PAL  Concept 
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PAL  Concept 
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Two  PAL  Efforts 
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PAL  Transition 
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Office  Assistant 


•  Communication 

-  E-mail,  IM,  Chat 

-  Phone  &  Fax 

•  Planning  &  Scheduling 

-  Project  Planning 

-  Travel  and  Event  Planning 

•  Information  Products 

-  Project  Status  Reports 

-  Technical  Documentation 

•  Meetings  &  Discussions 

•  Alerts  &  Reminders 


Command  Assistant 


Communication 

-  Messages,  IM,  Chat 

-  Radio,  Phone,  &  Corns 

•  Planning  &  Scheduling 

-  Ops  and  Intel  Planning 

-  Logistics  Planning 

•  Information  Products 

-  Commander’s  Daily  Brief 

-  C2  &  ISR  Web  Services 

•  Meetings  &  Discussions 

•  Alerts  &  Reminders 
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CALO  Project 


Cognitive  Assistant 
that  Learns  and 
Organizes  (CALO) 


SPA  WAR 
Systems  Center 
San  Diego 


“I  am  an  Al-based  Personal  Assistant 
dedicated  to  helping  you  manage  your  busy 
life.  Researchers  and  programmers  give 
me  knowledge,  but  more  and  more,  I  am 
learning  and  acquiring  new  knowledge  and 
abilities  on  my  own.  My  creators  call  this 
‘learning  in  the  wild’” 


Learning  in  the  Wild:  CALO  improves  by  adapting  to  the 
user  and  his  environment 


CALO  Functionality 


Cyber 

Manager 


Observe  &  Mediate 
Interactions 


Organize  &  Manage 
Information 


Perception 

Manager 


Schedule  & 
Organize  in  Time 


Acquire,  Allocate 
Resources 


Monitor  &  Manage 
Tasks 


Explanation  | 


Knowledge 

Manager 

I  Query  Mgr  | 

I  Update  Mgr 

Timeline  Mgr 


Prepare  Information 
Products 


Interpretation  | 


Interaction 

Manager 


CALO 


^  PrepPdCk 

1 

;  (Cak)-irisJ  CALO  Devek). 

^  Medmg  with  Adam  (6/2. 

[YesJ  ® 

^  CALO  Emai  Clustering  T 

(YesJ  ®  V 

J 


Supports  commanders  and  decision-makers  by  learning  from  and  adapting 
to  their  environment...  Allows  for  better  decisions  from  smaller  teams 


CALO  Task  Learning  Components 


Learning  By 
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By 
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By 

Discussion 
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Execution 
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Task 

Models 

Structured 

Process 
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Process 

Model 

Reasoning 
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Discovery 

Of  New 
Sources 

Acquiring 

Semi-structured 

Sources 

Meta-Knowledge 

About 

Sources 
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CW  Simulation  Exercise 


Co  mm  and  Wo  rid  (CW)  #5 

Hour 

April  11,  2005 

April  12,  2005 

April  13,  2005 

April  14,  2005 

April  15,  2005 

Hour 

0730 

(0745)  Assume  the 
Watch 

(0745)  Assume  the 
Watch 

0730 

0800 

Admin  /  Introducrtion 
(SSOSPAWAR) 

(0.5  hr> 

CALO  Training 
(SFU) 

(3.0  hrs) 

Scenario  Event  Two 
(CAP  Phase  1  (SA» 
(4.0  hrs) 

Scenario  Event  Four 
(CAP  Phase  II  (CA» 
(4.0  hrs) 

Busy  People  Transfer 
Learning  Experiment 
(MIT>SFU>SSC) 

(4.0  hrs) 

0800 

0830 

Privac:v  Form  Fteview 
(NPS) 

(0.5  hr> 

0830 

0900 

PAL/CALO  Overview 
Brief  (SFU> 

(1.0  hr) 

0900 

0930 

0930 

1000 

CAP  Training 
(SPAWAFt) 

(1.0  hr) 

1000 

1030 

1030 

1100 

Break 

Break 

1100 

1130 

1130 

1200 

(1215)  CAP  and 
Scenario  Training 
(SPAWAFt) 

(2.75  hrs) 

(1215)  Scenario 
Event  One 
(CAP  Phase  1  (SA)> 
(4.0  hrs) 

Break 

Break 

Break 

1200 

1230 

1230 

1300 

Scenario  Event  Three 
(CAP  Phase  II  (CA)> 
(4.0  hrs) 

Scenario  Event  Five 
(CAP  Phase  III  (COA)> 
(4.0  hrs) 

Wfap-up 
(2.0  hrs) 

1300 

1330 

1330 

1400 

1400 

1430 

1430 

1500 

WebCOP  Training 
(SYS) 

(1.0  hr) 

1500 

1530 

1530 

1600 

Wfap-up  and  FCeview 

1600 

1630 

Wfap-up  and  Fteview 

1630 

1700 

Daily  Wfap-up 
(SSC)SFU>SYS) 

Daily  Wfap-up 
(SSC/SFU/SYS) 

Exercise  Wfap-up 
(SSOSFU/SYS) 

1700 

V 

SPA  WAR 
Systems  Center 
San  Diego 

CW  #5 
Schedule 


During  CW  #5,  simulation  exercise  encompassed  the  first  three  phases 
of  the  CAP  model,  culminating  with  the  submission  of  course  of  action 
(COA)  alternatives  to  an  Executive  Decision-Maker 
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The  Information  System  for  CW 


SPA  WAR 
Systems  Center 
Sen  Dieao 
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Crisis  Action  Planning  (CAP)  Model 

•  WHAT  IS  A  CRISIS? 

-  Develops  rapidly;  threat  to  national  or  regional  interest 

-  May  require  a  military  response 

•  CAP  FEATURES 

-  Facilitates  rapid  exchange  of  information 

-  Requires  collaborative  &  concurrent  planning 

-  Analysis  of  situations  affecting  possible  COA 

-  Developing  valid  COA  and  selecting  the  best 

-  Coordinating  plans  and  orders  supporting  execution 

•  WHY  CAP? 


SPA  WAR 
Systems  Center 
San  Diego 


-  Time  constraints  prevent  deiiberate  planning 

-  No  existing  operation  pian  applies 

-  PACOM  as  exampie  of  COCOM  -  Over  70  crisis  events  pianned  and 
executed  in  past  10  years 


Designed  to  support  a  commander’s  efforts  to  develop,  analyze, 
select  and  implement  a  military  COA 
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The  CAP  Model 


V 

SPA  WAR 
Systems  Center 
San  Diego 


*  -  0PREP-3PCA 
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Operation  CROWN  JEWEL  Scenario 


SPA  WAR 
Systems  Center 
San  Diego 


Combatant  Commander  (COCOM) 
Commander,  Joint  Task  Force  (CJTF) 
(Mike  Howard) 


Virtual  Players  (e.g., 
messaging,  SPAM,  etc) 


Joint  Planning  Group  Director 
(Ethan  Day) 


Intelligence  Planner  (J2) 
(Irvin  Olson) 


Operations  Planner  (J3) 
(Dave  Jones) 


Logistics  Planner  (J4) 
(Les  Owens) 


Land  Planner  (G3) 

Maritime  Planner  (N3) 

Air  Planner  (A3) 

(George  Page) 

(Nelson  Hoffmann) 

(A.  Bailey) 

Other  Staff 
(Oscar  Felix) 


ii 


Other  U.S.  Military 
Commands  /  Forces 
(Max  Trovato) 


Coalition  Military 
Command  and  Forces 
(Cecil  Hargrove) 


Higher  Authority 
(SecDef,  CJCS, 
POTUS 
(Hugh  Grant) 


Supporting 
Commands,  Agencies 
(Steve  Smith) 


International  Agencies 
and  NGOs 
(Ingrid  Dogood) 


Regional  Embassies 
and  Missions 
(Reggie  Polsen) 


JPG  Cell  Personnel 


External  participant 


Non-Participant 
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Operation  CROWN  JEWEL  Scenario 


Task  Hierarchy  Matrix 


SPA  WAR 
Systems  Center 
San  Diego 


CAP  Event 

Scenario  Task 

Scenario  Day 

JPG  Director  Task 

Intel  Planner  J2  Task 

1.1  Assume  watch  and 
determine  known  facts,  current 
status,  rules  of  engagement 
(ROE)  and  conditions  of  Area  of 
Operations  (AO);  Commence 
Crisis  Action  Plannina  TCAP") 

1.1.1  Log  on  to  PC;  conduct  communications 
check  (e.g.,  e-mail,  chat),  accustomation 
with  IRIS  desktop  and  training  folders 

Day  1 

26  January 

(1.1.1-00)  Send  exercise 
commencement  e-mail  and  conduct  roll 
call  with  all  planners  via  chat 
(announce  requirement  for  Information 
Deliverable  -  Commander's  Daily 
Briefina  rJ2/J3/J4  as  co-leads') 

(1.1.1-02)  Activate  desktop  (e.g., 
acknowledge  commencement  e-mail; 
activate  chat  functionality;  open 
WebCOP) 

1.1.2  Determine  current  status  of  available 
forces 

Day  1 

26  January 

(1.1.2-00)  Task  planners  to  review 
assigned  forces  and  apportioned  forces 
for  planning  (e.g.,  force  tables, 
WebCOP) 

(1.1.2-02)  Review  IRIS  desktop,  CW 
Reference  Library  website  to  determine 
status  of  MNF  intelligence  resources; 
commence  research/collection  efforts 
for  Commander's  Daiiy  Briefing 

1.1.3  Conduct  intelligence  preparation  of 
the  AO;  determine  status  of  friendly 
intelligence  support 

Day  1 

26  January 

(1.1.3-00)  Task  J2  as  lead  planner  in 
developing  intelligence  summary  for 

AO 

(1.1.3-02)  Review  IRIS  desktop,  CW 
Reference  Library  website  to  determine 
intelligence  (including  Weather  and 
Topographical  information)  summary; 
engage  other  planners  accordingly  to 
solicit  information  requirements 

1.1.4  Provide  forces  available,  readiness 
status  of  MNF,  host  nation  political  situation, 
availability  analysis 

Day  1 

26  January 

(1.1.4-00)  Task  planners  to 
consolidate  information  summary 
depicting  apportioned  forces 
(availability),  MNF  forces  (availability), 
intelligence  summary 

(1.1.4-02)  Review  IRIS  desktop,  CW 
Reference  Library  website  to  determine 
status  of  MNF  intelligence  apportioned 
forces,  MNF  forces;  provide  status  to 

JPG  Director 

1.1.5  Provide  logistic  readiness  of  available 
forces,  logistic  impact  of  available  port 
infrastructure  in  operational  area,  status  of 
"hard  to  obtain"  materials  and  services 

Day  1 

26  January 

(1.1.5-00)  Task  J4  as  lead  planner  in 
developing  logistics  summary  for  AO 
(e.g.,  port  infrastructure,  materials, 
services) 

(1.1.5-02)  Support  J4  as  required 

1.1.6  Provide  communication  factors  and 
status  regarding  topographical  constraints, 
available  communications  resources, 
communications  readiness  of  available 

resources 

Day  1 

26  January 

(1.1.6-00)  Task  J6  (non-participant) 
as  lead  planner  in  developing 
communications  summary  for  AO 

(1.1.6-02)  Support  J6  as  required 

Each  event  linked  to  task  for  planners 
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Operation  CROWN  JEWEL  Scenario 


SPA  WAR 
Systems  Center 
San  Diego 


Information  Deliverables: 

•  Commander’s  Daily  Briefing  (CDB)  -  daily:  Status  update  on  forces 
(disposition,  availability),  space  (AO),  SOE 

•  Commander’s  Assessment  -  Phase  I:  Operational  Report  3  -  Pinnacle 
Commander’s  Assessment  (OPREP  3-PCA) 

•  Commanders  Estimate  -  Phase  III:  Mission  statement.  Situation 
Update,  COA  (three),  COA  constraints  and  additional  planning 
concerns 

•  “Free-play”  products:  administrative  requests,  requests  for  information 
(RFI)  germane  to  game-play 

•  Questionnaire  /  Event  Log  -  End  of  Scenario:  game  participants  to 
identify  potential  capabilities  of  PAL  technology  /  system  or  scenario 
improvement 
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Simulation  Domain  /  Experimentation 


SPA  WAR 
Systems  Center 
San  Diego 


WebCOP 
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Simulation  Domain  /  Experimentation 
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IRIS  (and  WebCOP) 
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Data  Collection 


SPA  WAR 
Systems  Center 
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•  Each  participant  notebook  computer  hosted  a  client 
that  logged  all  MS  Office  activities  to  a  central 
database  on  the  networked  CALO  server  via  the 
following  software  modules: 

-  FileSystemWatcher  (FSW) 

-  Internet  Explorer  (IE)  Plug-in 

-  MS  OUTLOOK 

-  Outlook  Plug-In 

-  Timeline  Server 

-  Timeline  Viewer 
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C2  Experimentation  Summary 
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•  In  support  of  data  collection  for  the  CALO  leaning 
algorithm  development: 

-  Exercise  generated  over  236,173  logged  events 

-  Scenario  scripts  were  sufficient  for  participants  to  perform 
essential  CAP  tasks 

-  Unclassified  exercise  provided  realistic  baseline  data  from  a 
military  domain  for  the  test  and  evaluation  of  CALO  software; 
overall  feeling  that  team  developed  cohesive  bond  and  became 
more  efficient  as  function  of  time  (expected  behavior) 

•  Good  attention  to  exercise  rhythm  and  sequence  of 
trigger  events  were  conducive  to  more  creative  and 
innovative  (realistic)  game-play 

•  Lack  of  collaborative  tools  (e.g.,  “white  board”)  was 
limitation  to  game  “realism” 
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Questions  &  Discussion 


23 


